Dual pen: OS/application pens

ABSTRACT

There is disclosed an interactive display system comprising an interactive surface for displaying an image and for receiving inputs from remote devices, the system being adapted to detect the presence of at least two remote devices proximate the interactive surface.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/445,107, filed Apr. 14, 2010, entitled, “INTERACTIVE DISPLAY SYSTEM,”which is a national phase of PCT Application No. PCT/GB2007/003855,filed Oct. 20, 2007, entitled, “INTERACTIVE DISPLAY SYSTEM,” whichclaims priority from GB Patent Application No. GB20060020047, filed onOct. 10, 2006, which are hereby incorporated by reference into thisapplication.

The invention relates to interactive display systems.

A typical example of an interactive display system is an electronicwhiteboard system. An electronic whiteboard system typically is adaptedto sense the position of a pointing device or pointer relative to aworking surface of the whiteboard, the working surface being aninteractive surface. When an image is displayed on the work surface ofthe whiteboard, and its position calibrated, the pointer can be used inthe same way as a computer mouse to manipulate objects on the display bypassing the pointer over the surface of the whiteboard.

A typical application of an interactive whiteboard system is in ateaching environment. The use of interactive whiteboards improvesteaching productivity and also improves student comprehension. Suchwhiteboards also allow use to be made of good quality digital teachingmaterials, and allow data to be manipulated and presented using audiovisual technologies.

A typical construction of an electronic whiteboard comprises providingan array or matrix of drive and sense coils behind or underneath theworking surface of the whiteboard to thereby form an interactivesurface, which coils interact with electromagnetic elements in thepointer device.

An electronic whiteboard system may also be associated with interactivetablets or graphic tablets which may be used by users of the system.Such graphic tablets may be constructed in a similar way—but on asmaller scale—to an electronic whiteboard. A grid array may be providedin conjunction with a working surface to form an interactive surface,and a user may work on such surface with a pointing device adapted tointeract with the interactive surface.

It should also be noted that graphic tablets may be used independentlyof a whiteboard system. For example they are known to be used as inputdevices for computer systems, where a graphic tablet may replace a‘mouse and mat’ arrangement.

In current whiteboard system arrangements, and in arrangements ofgraphic tablets for computer input use, the pointing device is typicallyprovided as a pen-style device. It has been typically considered that apen-style device is a natural choice for use with an interactive surfacesince it mimics the action of pen-on-paper that most users are familiarwith.

Typically such a pen may be provided with buttons which function in thesame way as the buttons of a computer mouse. In one arrangement, thedepression of a button is achieved by pressing the pen in an axialdirection toward the interactive surface.

In typical existing electronic whiteboard systems, the use of twopointing devices on the interactive surface simultaneously is notpossible. Generally two pointing devices would interfere with eachother. For example where the pointing devices is used as a ‘mousesubstitute’, moving a cursor around the display, the presence of twopointers on the interactive surface would cause the cursor to rapidlyfluctuate between the points of contact of the two pointing devices.

It is an aim of the invention to provide an improved interactive system.

The invention provides an interactive display system comprising aninteractive surface for displaying an image and for receiving inputsfrom remote devices, the system being adapted to detect the presence ofat least two remote devices proximate the interactive surface.

The system is preferably adapted to receive data in dependence on thedetection of the presence of either of the at least two remote devices.The system is preferably adapted to detect the presence of the at leasttwo remote devices proximate the interactive surface simultaneously.

The system is preferably adapted to receive data from both of the atleast two remote devices in the presence of the at least two remotedevices proximate the interactive surface simultaneously.

An allocated functionality enables a remote device to annotate adisplayed image. A plurality of remote devices may adapt a displayedimage.

An allocated functionality may enable the remote device to control agraphical user interface on the displayed image.

Preferably one of the at least two remote devices controls the otherremote device. The one of the at least two remote devices may enable anddisable the other remote device.

A functionality of the system may be provided in dependence uponsimultaneous data received from at least two remote devices. Thesimultaneous data received from at least two remote devicessimultaneously may control an application running on the interactivesystem. The functionality of the system may be dependent upon thedetection of movement of at least one of the remote devices. Thefunctionality of the system may be dependent upon the detection of adirection of movement of at least one of the remote devices.

An icon displayed on the screen and dragged toward the edge of thescreen may be displayed on the opposite side of the screen.

The interactive surface may be divided into at least two active areas,each remote device being associated with one active area, wherein anicon allocated for use by one user and displayed in one active areadragged towards the edge of the one active area by the one user is thendisplayed in the other active area and allocated for use by the otheruser.

An icon on the displayed image may be moved by selecting the icon byplacing a remote device proximate the interactive surface at a positionwhere the icon is displayed, and moving the pointer proximate thesurface in a given direction, wherein responsive thereto the icon isdisplayed in a different position of the interactive display other thanthe position where the remote device is located.

The icon may be displayed in the different position in dependence uponthe distance and/or speed the remote device is moved.

A plurality of active regions may be defined on the interactive surface,the icon being displayed on the image in an active region in which thepointer is currently being used.

The invention provides an interactive system comprising an interactivesurface for displaying an image and for receiving inputs from aplurality of remote devices, the interactive surface defining at leasttwo active areas, each active area being associated with at least oneremote device.

Data from a remote device may be processed only when the remote deviceis detected in an active area with which it is associated.

Multiple active areas may be defined within an application. Multipleactive areas may be defined within a work surface of an application.

The invention provides an interactive system comprising an interactivesurface for displaying an image and for receiving inputs from aplurality of remote devices, the system being adapted to manipulate animage displayed on the interactive surface in dependence on datareceived from the plurality of remote devices.

The system may be adapted to manipulate the image responsive todetection of movement of one of the remote devices. The system may beadapted to manipulate an image responsive to detection of the pluralityof remote devices selecting the image.

The image may be rotated responsive to detection of a first remotedevice positioned on the displayed image and stationary, and detectionof a second remote device positioned on the displayed image and moving,the rotation being about an axis defined by the position of the firstremote device.

The image may be rotated responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the rotation beingdetermined in dependence on detected movement of either the first orsecond remote devices.

The image may be resized responsive to detection of a first remotedevice positioned on the displayed image and stationary, and detectionof a second remote device positioned on the displayed image and moving,the image size being decreased responsive to the distance between thefirst and second remote devices decreasing and the image size beingincreased responsive to the distance between the first and seconddevices increasing.

The image may be resized responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the image size beingdecreased responsive to the distance between the first and second remotedevices decreasing and the image size being increased responsive to thedistance between the first and second devices increasing.

The image may be stretched responsive to detection of a first remotedevice positioned on the displayed image and stationary, and detectionof a second remote device positioned on the displayed image and moving,the image size being stretched responsive to the distance between thefirst and second remote devices increasing.

The image may be compressed responsive to detection of a first remotedevice positioned on the displayed image and stationary, and detectionof a second remote device positioned on the displayed image and moving,the image size being compressed responsive to the distance between thefirst and second remote devices decreasing.

The image may be resized responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the image size beingcompressed responsive to the distance between the first and secondremote devices decreasing and the image size being stretched responsiveto the distance between the first and second devices increasing.

The image may be moved responsive to detection of a first remote devicepositioned on the displayed image, and detection of a second remotedevice positioned on the displayed image, the movement being determinedin dependence on detected movement of either the first or second remotedevices.

The image may be separated responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the separation beingdetermined in dependence on detected movement of either the first orsecond remote devices. The image may be separated at a point midwaybetween the detected starting positions of the two remote devices.

Separate images may be joined responsive to detection of a first remotedevice positioned on a first displayed image, and detection of a secondremote device positioned on a second displayed image, the joining beingdetermined in dependence on detected movement of either or both of thefirst or second images such that the images are positioned within apredetermined distance of each other. Said joining may create a singleimage or a connection between the two images.

The image may be duplicated responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the duplication beingdetermined in dependence on a detected dragging movement of either thefirst or second remote devices and providing a duplicate image at alocation to which the dragged remote device is positioned.

The image may be cropped responsive to detection of a first remotedevice positioned on the displayed image, and detection of a secondremote device positioned on the displayed image, the two detectedpositions defining opposite corners of a cropping rectangle the size ofwhich is adjusted in dependence on movement of either or both of theremote devices to define a cropping area for the image.

A distance may be measured responsive to detection of a first remotedevice positioned on the interactive display, and detection of a secondremote device positioned on the interactive display, the detectedpositions being used to determine the distance between the two remotedevices.

The invention provides an interactive system comprising an interactivesurface for displaying an image and for receiving inputs from at leastone remote device, the system being adapted to manipulate an imagedisplayed on the interactive surface in dependence on data received fromthe remote device, wherein responsive to detection of the remote deviceselecting an image and dragging it toward the edge of the interactivesurface, the image is displayed on the opposite edge of the interactivesurface.

There may be provided a plurality of remote devices, each remote devicebeing associated with one or more active areas, wherein responsive to animage displayed in one active area and associated with one remote devicebeing dragged by that remote device to the edge of that active area, theimage is displayed in another active area and re-associated with anotherremote device associated with that active area.

The invention provides an interactive system comprising an interactivesurface for displaying an image and for receiving inputs from at leastone remote device, a displayed image being associated with a remotedevice, wherein the position of the displayed image on the interactivesurface is dependent upon a current position of the associated remotedevice on the interactive surface.

The invention provides an interactive system comprising an interactivesurface for displaying an image and for receiving inputs from at leastone remote device, the system being adapted to detect a predeterminedsequence of movements of the remote device, and responsive thereto anoperation is enabled.

The predetermined sequence of movements may correspond to a usergesture. The predetermined sequence of movements is detected between aset of points.

The invention provides a method in an interactive display systemcomprising an interactive surface for displaying an image and forreceiving inputs from remote devices, the method comprising detectingthe presence of at least two remote devices proximate the interactivesurface.

The method may comprise the step of allocating a functionality to eachof the at least two remote devices. The step may allocate each remotedevice at least one different functionality. The step may allocate eachremote device at least one common functionality.

The method may comprise the step of, when a remote device is detected inan active region of the interactive surface not defined for that remotedevice, not processing any data from that remote device.

The method may define a physical area as an active region of theinteractive surface. The method may define an icon on the displayedimage or a graphical user interface on the displayed image as an activeregion of the interactive surface.

The method may comprise the step of dividing the interactive surfaceinto at least two active areas, each remote device being associated withone active area, wherein an icon allocated for use by one user anddisplayed in one active area dragged towards the edge of the one activearea by the one user is then displayed in the other active area andallocated for use by the other user.

Responsive to the step of an icon being moved toward the edge of oneactive area, there may be displayed the icon on the opposite side ofanother active area.

The method may comprise the step of moving an icon on the displayedimage by selecting the icon by placing a remote device proximate theinteractive surface at a position where the icon is displayed, andmoving the pointer proximate the surface in a given direction, whereinresponsive thereto the icon is displayed in a different position of theinteractive display other than the position where the remote device islocated.

The method may comprise the step of moving the displayed icon to an edgeof the displayed image corresponding to the direction of movement of theremote device.

The method may comprise the step of displaying the icon in the differentposition in dependence upon the distance the remote device is moved.

The method may comprise displaying the icon in the different position independence upon the speed with which the remote device is moved.

The method may display the position of an icon in the image independence upon a current detected location of a remote device for whichthe icon is an active icon.

The method may comprise the step of displaying the icon at an edge ofthe image nearest to the current detected location of the remote device.

The method may comprise the step of defining a plurality of activeregions on the interactive surface, the icon being displayed on theimage in an active region in which the pointer is currently being used.

The invention provides a method in an interactive system including aninteractive surface for displaying an image and for receiving inputsfrom a plurality of remote devices, the method comprising defining atleast two active areas of the interactive surface, each active areabeing associated with at least one remote device.

The method may process data from a remote device only when the remotedevice is detected in an active area with which it is associated.

The active area may be a physical area of the interactive surface. Theactive area may be an icon or graphical user interface displayed on theinteractive surface. The method may define multiple active areas withinan application or multiple active areas within a work surface of anapplication.

The invention provides a method in an interactive system comprising aninteractive surface for displaying an image and for receiving inputsfrom a plurality of remote devices, comprising the step of manipulatingan image displayed on the interactive surface in dependence on datareceived from the plurality of remote devices.

The method may comprise the step of manipulating the image responsive todetection of movement of one of the remote devices. The method maycomprise manipulating an image responsive to detection of the pluralityof remote devices selecting the image.

A distance may be measured responsive to detection of a first remotedevice positioned on the interactive display, and detection of a secondremote device positioned on the interactive, display, the detectedpositions being used to determine the distance between the two remotedevices.

The invention provides a method in an interactive system comprising aninteractive surface for displaying an image and for receiving inputsfrom at least one remote device, comprising the step of manipulating animage displayed on the interactive surface in dependence on datareceived from the remote device, wherein responsive to the step ofdetection of the remote device selecting an image and dragging it towardthe edge of the interactive surface, further comprising the step ofdisplaying the image on the opposite edge of the interactive surface.

The image may be a graphical user interface icon. The interactivesurface may comprise a plurality of active areas, comprising the step ofdisplaying a dragged image in a different active area.

There may be provided a plurality of remote devices, each remote devicebeing associated with one or more active areas, comprising the step of,responsive to an image displayed in one active area and associated withone remote device being dragged by that remote device to the edge ofthat active area, displaying the image in another active area andre-associating the image with another remote device associated with thatactive area.

The invention provides a method in an interactive system comprising aninteractive surface for displaying an image and for receiving inputsfrom at least one remote device, a displayed image being associated witha remote device, comprising the step of the position of the displayedimage on the interactive surface being dependent upon a current positionof the associated remote device on the interactive surface.

The invention provides a method in an interactive system, the systemcomprising a surface for displaying an image and adapted for receivinginputs from at least one remote device, the method comprising the stepsof detecting a predetermined sequence of movements of the remote device,and responsive thereto enabling an operation.

The predetermined sequence of movements may correspond to a usergesture. The predetermined sequence of movements may be detected betweena set of points. Each point may be an area within which presence of theremote device can be detected.

The method may comprise the step of beginning and ending each sequenceof movements at the same point.

The method may comprise the step of enabling the operation responsivefurther to detection of the remote in a region above the interactivesurface and not in contact with the interactive surface.

The points may comprise a start point and at least one further point.The points may comprise a central point and a set of further pointspositioned radially around the central point.

The set of further points may be positioned radially around thecircumference of a circle having a centre at the central point.

The predetermined sequence of movements may be between any number of thepoints, any number of times. The predetermined sequence of movement maydefine a shape.

The operation may be enabled responsive to the speed of thepredetermined sequence of movement.

The method may further comprise displaying the points on the surface fordisplaying an image.

The surface may be an interactive surface for receiving inputs from atleast one remote device.

A displayed image may be an icon representing a graphical userinterface. A displayed image may be an image displayed in anapplication. A displayed image may be a photograph.

The invention is described by way of example with reference to theaccompanying Figures, in which:

FIG. 1 illustrates an example of an interactive display system;

FIG. 2( a) illustrates an example of the functional elements of awhiteboard apparatus arrangement of an interactive display system;

FIG. 2( b) illustrates an example functional structure of a pointingdevice for use with the whiteboard apparatus arrangement of FIG. 2( a);

FIG. 3 illustrates a portion of a grid array of an interactive surfaceassociated with the whiteboard apparatus arrangement of FIG. 2( a);

FIG. 4 illustrates an example of the functional elements of FIG. 2( a)further adapted to support embodiments of the invention;

FIG. 5 illustrates an example flow chart for operation of the functionalelements of FIG. 4;

FIG. 6 illustrates a software architecture in a preferred embodiment;

FIG. 7 illustrates the interactive surface of an exemplary interactivesystem;

FIGS. 8 to 15 illustrate examples of gestures in preferred embodimentsof the invention;

FIG. 16 illustrates a gesture setting application in an embodiment; and

FIG. 17 illustrates a gesture setting application in an embodiment.

Referring to FIG. 1, an exemplary interactive display system comprises awhiteboard assembly arrangement generally designated by referencenumeral 102, a computer 107 having an associated display 106, and aprojector 104. The computer 107 is connected to the whiteboard assemblyarrangement 102 via a communication link 108, and to the projector 104via a communication link 110. The projector 104, which may be fitted toa ceiling of a room such as a classroom, receives signals from thecomputer 107 which are translated into corresponding projection imagesfor projection onto a display surface 114 of the whiteboard assemblyarrangement 102.

The image projected on the display surface 114 of the whiteboardassembly arrangement 102 may be the same as that displayed on the screen106 of the computer 107.

The interactive display system also includes one or more pointingdevices or pointers, as represented by pointing device 112, whichcooperate with the whiteboard assembly arrangement 102. The pointingdevice 112 is moved across the display surface 114 of the whiteboardassembly arrangement 102, in contact with or close to the surface. Theposition of the pointing device 112 relative to the display surface 114of the whiteboard assembly arrangement 102 is, in one type ofarrangement, detected electronically by means of a wire grid embeddedbeneath the display surface 114. The pointing device 112 may be movedaround the display surface 114 to write on the display surface, forexample, or to highlight images displayed on the display surface. Theuse of such a pointing device in combination with a whiteboard assemblyarrangement is well-known to one familiar with the art.

Using methods known in the art, the pointing device 112 can function inthe same way as a computer mouse. The pointer may be provided withbuttons or such like which may be depressed, to provide for functionaloperations in much the same way as buttons may be provided on a computermouse. For example, by depression of a button a displayed icon overwhich the pointing device 112 is positioned may be selected. Forexample, by depression of a button the functional operation of thepointer may change from a pen to an eraser.

In general, the movement of the pointing device 112 across the displaysurface 114 is detected by the embedded grid array, and such movementtranslated to be superimposed on the displayed image, such that thedisplayed image projected by the projector 104 is adapted to display anyrequired action associated with the pointing device, as is known in theart.

The structure of the whiteboard assembly arrangement and the pointingdevice for operation of the interactive display system may be one ofseveral different implementations. In a preferred arrangement thewhiteboard assembly arrangement 102 includes a grid portion behind thedisplay surface, which comprises two sets of wire loops arrangedorthogonally to each other. The pointing device 112 is adapted to inducea current in the wire loops which can be used to determine the positionof the pointing device 112. In a particularly preferred arrangement thepointing device 112 is a passive electromagnetic device: a drive gridinduces a current in the pointing device, which in turn induces acurrent in a sense grid. The operation of such an arrangement isdiscussed further below with reference to FIG. 3 in combination withFIG. 2( a).

Electronic control circuitry is preferably provided within thewhiteboard assembly arrangement 102 for processing signals generated bycooperation of the wire grid beneath the display surface and thepointing device, and to thus determine the position of the pointingdevice and information corresponding to any provided buttons on thepointing device, being selected.

With reference to FIG. 2( a), there is shown an exemplary overview ofthe functional elements of a preferred whiteboard assembly arrangement,which may be provided by control circuitry associated with whiteboardassembly arrangement 102.

The exemplary whiteboard assembly arrangement 102 includes a drive grid202 and a sense grid 204.

The drive grid 202 consists of a first plurality of conducting coilsarranged in a first orientation and a second plurality of conductingcoils arranged in a second orientation, the second orientation beingorthogonal to the first orientation. One set of coils, hereinafterreferred to as the X drive coils, provides a set of X-axis drive coils,and the other set of coils, hereinafter referred to as the Y drivecoils, provides a set of Y-axis drive coils.

The sense grid 204 consists of a first plurality of conducting coilsarranged in a first orientation and a second plurality of conductingcoils arranged in a second orientation, the second orientation beingorthogonal to the first orientation. One set of coils, hereinafterreferred to as the X sense coils, thus provides a set of X-axis sensecoils, and the other set of coils, hereinafter referred to as the Ysense coils, provides a set of Y-axis sense coils.

The sense grid 204 comprises a balanced array or matrix of conductingcoils laid side by side, each coil being paired with an identical butoppositely wound coil, the coils being inter-connected so as to give amulti-phase output signal. The pattern of inter-connection is repeatedmany times over the area of the whiteboard, with each complete patternbeing referred to commonly as a ‘pitch’.

The sense grid 204 has two separate and independent such arrays ofcoils, which are placed orthogonal to each other to permit positionsensing in perpendicular X and Y axes. The pattern of coils ispreferably produced by wiring of a conductive material.

The drive grid 202 is also formed as two orthogonal arrays or matrices,for driving in perpendicular X and Y axes, and may be fabricated by thesame techniques as is the sense grid. The drive grid comprisesindividual coils laid side-by-side which coils are nominally of a pitchor smaller in width.

The drive grid is connected to receive drive signals from both an X-axisdrive multiplexer 206 and a Y-axis drive multiplexer 208. The X-axis andY-axis drive multiplexers 206 and 208 provide excitation current to onesof the X and Y drive coils respectively. The drive signals aresubstantially sinusoidal, and are preferably generated by a programmablesignal source which is locked to a stable reference frequency.

A drive grid signal generator 210 generates drive signals to each of theX-axis and Y-axis drive multiplexers 206 and 208.

The operation of the X-axis and Y-axis drive multiplexers is controlledby a processor 212, which provides a control signal to each of theX-axis and Y-axis drive multiplexers and the drive grid signalgenerator.

The drive grid signal generator 210 is preferably coupled to a poweramplifier which boosts the available current for the drive signals. Thedrive grid signal generator 210 also provides clock signals as an outputthereof to demodulation circuitry.

The pointing device 112 is, in a preferred embodiment, a devicecontaining a tuned circuit. When the drive multiplexers drive analternating current into the X or Y drive coils of the drive grid, theassociated changing magnetic field induces a voltage signal in the tunedcircuit of the pointing device. The resultant current induced in thepointer then causes a magnetic field which induces a voltage signal inthe X and Y sense coils of the sense grid.

An example of the functional elements of a prior art pointer isdescribed in more detail with reference to FIG. 2( b). FIG. 2( b) showsschematically a circuit of a pointer for use in combination with thewhiteboard apparatus arrangement of FIG. 2( a). The pointer comprises anLC tuned circuit including a coil 250 and a capacitor 260. In parallelwith the tuned circuit are one or more switched resistors. In FIG. 2( b)there is shown a first series combination of switch 262 a and resistor264 a in parallel with the tuned circuit, and a second seriescombination of switch 262 b and resistor 264 b. In the prior art whenthe pointer is in the form of a stylus or pen, the resistors may beswitched either axially by depressing the stylus tip against the worksurface, or radially using buttons located on the side of the pen.

In use, an alternating magnetic field at, for example, a frequency f₁caused by the excitation current in a coil of the drive grid acts on thetuned circuit of the pointer, the resonant frequency of which is set toapproximately f₁. This causes the tuned circuit to resonate, and themagnetic field produced by the coil 250 induces a voltage signal intothe sense grid.

In general, the drive grid produces an electromagnetic field whichcouples into the coil of the tuned circuit. This electromagnetic fieldmay induce a current or voltage in the coil of the tuned circuit. Thecoil of the tuned circuit produces an electromagnetic field whichcouples into the sense grid. This electromagnetic field may induce acurrent or voltage in the sense grid.

The resistors may be switched into the circuit so as to alter theQ-factor of the LC tuned circuit.

A sense multiplexer 214 is connected to receive output signals from thesense grid 204. Current induced in ones of the X and Y sense coils isdetected at the sense multiplexer.

The sense multiplexer 214 provides an output which is connected to aninput of a synchronous demodulator 216. After the sensed voltage signalsare received by the sense multiplexer 214 they are thus de-modulated inthe synchronous demodulator. The purpose of the synchronous demodulatoris to reject any extraneous noise and unwanted background signals. Thedemodulation clock for the synchronous demodulator is derived from thedrive grid signal generator 210. The demodulation clock circuitryprovides both phase and quadrature data to be discerned from the sensedsignals.

The digital signals produced at the output of the synchronousdemodulator 216 are then output to the processor. The processorpreferably processes such signals to calculate the position of thepointer. The calculated position information is then further output, viaan output interface 220, to a host device such as computer 106 of FIG.1.

The processor 212 generates control signals on outputs to each of thedrive grid signal generator 210, the sense multiplexer 214, thesynchronous demodulator 216, and each of the X and Y drive multiplexers206 and 208.

The operation of the whiteboard assembly arrangement 102 and thepointing device 112 are now described in more detail.

The X and Y drive coils are superimposed to the whiteboard assemblyarrangement 102 display surface 114. The drive coils of the drive gridcan be selected individually and, if required, in a random fashion bythe drive multiplexers under the control of the processor. When drivingthe X drive grid, the Y sense grid is connected to the synchronousdemodulator via the sense multiplexer. Conversely when driving the Ydrive grid, the X sense matrix is connected to the synchronousdemodulator via the sense multiplexer.

The balanced array of each of the X sense coils and Y sense coils issuch as to produce a nominal null in the sensed signal when therespective orthogonal drive coil is excited and no pointing device ispresent. This is due to the fact that any signal induced in one of theclockwise sense coils directly by the orthogonal drive coil will beinduced in an equal and opposite sense in the corresponding counterclockwise sense coil. However when the pointer which is excited by adrive coil at its resonant frequency is placed in proximity to a sensecoil it retransmits a magnetic field which induces a voltage signal inthe sense coils according to its position relative to the sense coils.

FIG. 3 shows an arrangement of a Y portion of the sense grid and a Yportion of the drive grid in an exemplary embodiment. The sense grid,for the purposes of illustration in this embodiment, is of a four-phasenon-overlapping type. The technique of operation is suited to many typesof commonly used grid or matrix topologies and is particularly suited totopologies in which sense coils are arranged so that there is nominallya null sense voltage when no pointer is present, as discussed above.

The Y sense grid has an interconnected pattern which repeats severaltimes across the display surface, each repeat being commonly referred toas a pitch. The number of pitches needed in any arrangement is dependentupon the width of each pitch and the size of the display surface. Thesense grid permits the processor to determine with high resolution theposition of the pointer within the pitch. This is achieved as follows.

Four phase signals from the sense coil are amplified and demodulated bythe synchronous demodulator to generate DC voltage levels. The DCvoltage levels are proportional to the amplitude of the AC signal whichis being demodulated from the sense coils. The DC voltage levels arethen converted into digital values by an analogue to digital converter(not shown) and are sent to the processor. The processor undertakes avector summation on the four numbers (representing the four differentlyphased coils) and from this the exact position of the pointing devicewith respect to the four coils, i.e. within a particular pitch, isdetermined.

However the signal from the sense matrix alone cannot determine theabsolute position of the pointer, since the processor cannot know fromthis information alone in which pitch the pointer is inducing thesignals.

In order to determine the pitch in which the signals from the pointingdevice are induced, it is necessary to excite selectively theappropriate drive coils. The number of drive coils is equal to orgreater than the number of pitches. For example to determine the pitchin the X axis the processor selectively energises the X drive coils anddetermines the peak amplitude and phase of the signals induced in the Ysense coils for each selected X drive coil. From this amplitude andphase information the X pitch is determined.

Conversely by selecting the Y drive coils and analysing the X sensecoils the Y pitch is determined.

There has thus been described the functional structure and operation ofan exemplary interactive display system including a whiteboard assemblyapparatus.

As described in the foregoing discussion, in a typical prior artarrangement a pen-style device is used as a pointer or pointing devicefor interacting with the interactive surface of the whiteboard surface.However other types of devices are also provided as pointing devices.

In accordance with embodiments of the present invention theabove-described interactive display system is adapted to incorporatemore than one pointing device which may be used in co-operation with theinteractive surface at any one time. In the following description anexemplary arrangement will be referred to in which two pointing devicesare provided. However it should be understood that the principlesdescribed apply to the provision of any number of a plurality ofpointing devices, and is not limited to two.

With reference to FIG. 4 an exemplary arrangement of the functionalelements of the whiteboard apparatus arrangement of FIG. 2( a), modifiedto support the provision of multiple pointing devices, is described. Forthe purposes of example an implementation with two pointing devices isdescribed.

In order to distinguish between the two pointing devices in use at thesame time on the interactive surface, each of the pointing devices isprovided with a unique nominal frequency. That is the frequency of eachpointing device is within a pre-allocated unique frequency band. For thepurposes of this example it is assumed that one pointing device isallocated a nominal frequency f₁ and another pointing device isallocated a nominal frequency f₂.

The functional elements of the whiteboard apparatus arrangement operatein a manner similar to that described hereinabove, for each distinctnominal frequency. The drive grid signal generator 210 is adapted suchthat it can generate a nominal frequency of either f₁ or f₂. In thearrangement of FIG. 4 the drive grid signal generator 210 is shown togenerate two distinct frequencies on two distinct signal lines. Inpractice, however, the drive grid signal generator 210 may be aprogrammable signal generator, having a single output, on which line oneof frequencies f₁ or f₂ is generated at any one time. In the arrangementof FIG. 2, a selection circuit 402 is connected to receive the two linescarrying the frequencies f₁ and f₂, and provide one of such frequencieson an output line as a drive frequency denoted by f_(d). The selectionblock 402 operates under the control of a control signal CON from theprocessor 212. This control signal may also provide to the drive gridsignal generator 210.

The processor 212 interfaces with a timer device 404. The principle ofoperation is that the drive grid or drive array of the interactivesurface is driven for a period of time at the nominal frequency f₁, andthen driven for a subsequent period of time at the nominal frequency f₂.This alternative driving is then repeated. Preferably the drive grid isdriven at the two frequencies for an equal amount of time. In thearrangement of FIG. 4 the timer 404 is provided to facilitate thisoperation. The processor 212 sets the timer 404 using a control signalSET. The processor 212 then monitors the current time provided as asignal TIME from the timer 404. When the appropriate time has elapsedthe processor 212 sets the control signal CON to switch the select block402 and thus switch the drive frequency.

As in the typical interactive display systems described hereinabove withreference to FIG. 2( a), the drive frequency f_(d) is provided to eachof the X drive block 206 and Y drive block 208. The X drive block 206and the Y block 208 are controlled by the processor 212, via respectivecontrol signals CON_(dx) and CON_(dy). Thus the driving of the drivegrid is carried out in a similar manner to that described earlierherein, the difference being that the drive frequency alternates betweenthe two nominal frequency values.

As described in relation to the arrangement of FIG. 2( a) hereinabove,the processor 212 receives a signal from the synchronous demodulator216. Thus the processor 212 is able to detect when a signal is providedby the synchronous demodulator 216 indicating the presence of a pointingdevice. If such a signal is detected, this indicates that a pointingdevice having the current drive frequency is present on the interactivedisplay. The drive grid signal generator 210 preferably provides anindication to the processor of the current selected frequency by way ofa signal SEL_(f), and therefore when the processor 212 processes thesignal from the synchronous demodulator in order to determine theposition and status of the pointing device, this information can bedetermined as being associated with the specific pointing device.

The operation of the interactive display system in accordance with theseprinciples is now further explained with reference to the flow diagramof FIG. 5.

In a step 502 a time period is set. This may be a default time periodwhich is set in dependence upon the number of pointers intended to be inuse. A nominal time period will be set, and subdivided in accordancewith the number of pointers to be used. Thus where two pointers are tobe used, this time period will be divided by two, where three pointersare to be used this time period will be divided by three etc. Theappropriately divided time period is then set as a time value and storedunder the control of the processor 212.

In a step 504 an integer value i is set to 1. For the purposes of ageneral example, it is assumed that in general there may be n pointersin use in the system at any one time.

Thereafter in a step 506 the drive grid is driven at a frequency f_(i).As i is currently set to 1, the drive grid is driven at frequency f_(i).

In a step 508 it is determined whether a pointer is detected. This isindicated by the synchronous demodulator output being processed by theprocessor 212 as is known in conventional systems.

In response to a pointer being detected in step 508, based on a signalfrom the synchronous demodulator 216, the processor processes thereceived information in a step 510 to determine the location of thepointer (i.e. the X-Y location), and in step 512 determine the status ofthe pointer (button depressions) if appropriate.

If no pointer is detected in step 508, or alternatively on completion ofsteps 510 and 512 where a pointer is present, in a step 518 it isdetermined whether the set time has elapsed. If the set time has notelapsed then the process returns to step 506 and the process repeats. Ifthe set time has elapsed then in a step 514 it is determined whether iis currently set at the value n. If the value i is currently set at thevalue n then all available frequencies have been scanned, and theprocessor returns to step 504 and i is reset to 1.

If in step 514 it is determined that i is not equal to n, then thereremain other frequencies to be processed in the cycle. In a step 516 thevalue of i is incremented, for example to 2, and then the drive grid isdriven at the new frequency f_(i) (e.g. f₂) in step 506 and the abovedescribed steps repeated.

Thus in general in a first time period the drive apparatus is set to afrequency associated with one of multiple pointing devices and the X-Yposition and status of that pointing device is determined. In the nextperiod the frequency is set to another device frequency and again theposition and status determined.

The implication of this sequential operation is that the update rate foreach pointing device is directly dependent upon the number of devicespresent. In a practical system all the pointing devices potentially inuse are not in proximity with the surface all of the time, and thesystem may be controlled such that if no pointing device of a particularfrequency is detected in a particular cycle or in a particular number ofcycles, the sampling rate for that frequency is reduced until a deviceoperating at that frequency is found, at which point the sampling ratefor that frequency returns to the highest rate. As discussed hereinabovethe highest sampling rate is preferably the total maximum rate dividedby the number of devices in use. In certain situations, however, someparticular devices may be associated with an application which does notrequire the highest data rate, and the system may adapt the samplingrate for any given frequency to optimise the sampling rate of particulardevices to maximise their performance. This may involve providing asampling rate higher than the nominal sampling rate for certain devices,whilst providing a sampling rate lower than the nominal sampling ratefor other devices.

It should be noted that it is not essential that the various frequenciesbe sampled alternately. The described technique is merely exemplary.

In a preferred arrangement of the invention the drive grid signalgenerator 210 is adapted such that it may lock onto the centre frequencyof a coil provided in a pointing device operating at a nominalfrequency. Thus the drive grid signal generator may initially drive thegrid at a frequency f₁, and then lock on accurately to the actualfrequency of a pointing device having a nominal frequency of f₁.

It should be further noted that each individual pointing device mayoperate in accordance with the operation of conventional pointingdevices. In a preferred arrangement this involves the detection of thestatus of a pointing device in dependence upon the alteration of its Qfactor. The depression of a button on a pointing device, for example,alters the Q factor of the device which can then be detected todetermine its status. Thus in the preferable arrangement the pointingdevice always operates at its nominal frequency, with its status beingdetermined by adjustment of its Q factor.

The data for each pointing device can be processed separately, and thisinformation may be passed to any application that is designed to processthe data for a particular pointing device.

In general, for an interactive display system, a pointing device has twopurposes. A first purpose is to control system functionality, and asecond purpose is to allow use of applications. In order to controlsystem functionality a pointing device is able to operate as aconventional ‘mouse’ device when connected to a computer. Thus thepointing device is able to control the operating system such as aWindows operating system, of the computing system of which theinteractive system forms an interface.

To support use in various applications, the pointing device acts as aninput device. For example in a drawings package the pointing device maybe used to draw lines etc. on a ‘work pad’ surface displayed.

In a preferred arrangement of the present invention, for use ininteractive systems, a first pointing device is defined as a systempointing device, and one or more further pointing devices are defined asmulti-mode pointing devices. These pointing devices alternatively may bereferred to as a ‘system pen’ and a ‘private pen’ respectively.

A system pointing device is able to control the operating system, in away that a computer ‘mouse’ does, and preferably is also used forinteracting with various applications. A multi-mode pointing device is,however, not able to control the operating system. A multi-mode pointingdevice is able to interact with an application which is configured formulti-pointing device use. In general a system pointing device caninteract with all applications, whilst a multi-mode pointing device caninteract only with multi-pointing device specific applications.

It should be understood, for the purposes of this description, that a‘system pointing device’ is a pointing device that can be used in anystandard application, as well as applications supporting multiplepointing devices. A ‘multi-mode pointing device’ preferably cannotinteract with standard applications, but can preferably only interactwith applications supporting multiple pointing devices.

The system pointing device preferably has overall control of theinteractive system. This preferably includes the system pointing devicehaving control of any multi-mode pointing device. Specifically thesystem pointing device may be able to enable or disable any multi-modepointing device, to control the use of the multi-mode pointing device.When the system pointing device disables the multi-mode pointing device,the apparatus of FIG. 4 may be controlled such that the drive grid isnever driven at the nominal frequency of that multi-mode pointing deviceand thus that multi-mode pointing device is never detected regardless ofits presence on the interactive surface. Preferably an icon may bedisplayed on the interactive surface which the user of the systempointing device may ‘click-on’ to enable/disable any multi-mode pointingdevice.

It is emphasised that in accordance with the present invention there isprovided a technique which enables a plurality of pointing devices, ofany number two or greater, to be used together in an application. Forthe purpose of clarity, in the following examples are presented of a twopen operation. However this limitation of the multi-pen principle to adual-pen example is purely illustrative.

On the basis of a ‘dual-pen’ example, the following operationalscenarios can be envisaged:

-   1) A teacher may use the system pointing device only. The teacher    controls the operating system and standard applications. The teacher    never launches an application which requires a dual input, or does    not use dual input functionality of other applications. This is, in    effect, a mode of operation equivalent to use of a typical    conventional whiteboard system.-   2) A teacher uses the system pointing device primarily. The teacher    can use the system pointing device to launch a dual application. The    teacher may then use both the system pointing device and the    multi-mode pointing device in the dual application. The teacher uses    the system pointing device to close the dual application.-   3) A teacher uses the system pointing device primarily. The teacher    can use the system pointing device to launch a dual application. The    student is then given a multi-mode pointing device to work alongside    the teacher in the dual application. Using the system pointing    device, the teacher can discretionally switch the input from the    multi-mode pointing device on or off. The teacher uses the system    pointing device to close the dual application. The student cannot    use the multi-mode pointing device to operate anything outside of    the dual-application.-   4) A teacher uses the system pointing device to launch a dual    application. The system and multi-mode pointing devices are then    given to two students to operate the application. The teacher can    provide overriding control using a standard ‘mouse’ attached to a    computer associated with the interactive system.

Allowing two users to interact simultaneously with an application bringsnew problems to consider. Many scenarios can be considered, such as twopeople writing at the same time, both moving objects into position, oneperson annotating in a flipchart whilst the other person drags imagesfrom the resource library. The possibilities are almost limitless.

Example dual-input applications may include:

-   -   Dual Draw    -   Dual Colouring In    -   Dual Jigsaw    -   Dual Concept Mapper    -   Dual Image Editor    -   Dual Image Sorter    -   Dual Musical Instrument Player    -   Dual Drum Kit    -   Dual Note Taker    -   Dual Connect 4    -   Dual Maths Challenge    -   Dual Paddle Football

It is advantageous to design a dual or multi-user application such thatthe participants are kept mainly to their respective sides of the board.This helps in two respects. Firstly it prevents users getting in eachothers way, and secondly it allows the audience to more clearly observethe central area of the whiteboard where the activity is taking place.

Consider a Jigsaw application. One person might drag a jigsaw piece fromthe left-hand side to the right-hand side of the board whilst anotherperson is dragging a piece the other way. In this situation the twopeople may have to step under and over each other whilst trying to keepthe pointing devices on the board in order to maintain the dragoperations.

This problem can be overcome by ensuring that each jigsaw piece isinitially arranged such that it is randomly positioned in its correcthalf of the board, i.e. that half in which it will be located on thefinished jigsaw. Also, the careful layout of pieces will help tominimise the time spent by the participants in the central area of theboard, hence allowing the audience to see the formation of the jigsawmore clearly.

An example application for multi-pen use may be a technical drawingsurface. In this one pen is used to move and ‘pin’ a displayed rulerinto position on a grid, whilst the other pen is used to draw straightlines along the edge of the ruler.

The ideal position for tools and toolboxes is down either side of theboard. Some tools can be for the system pointing device only (e.g. fileopen/save commands); others may be for the multi-mode pointing device;some are for both.

With a system pointing device it is possible to feed data into theapplication from two streams; the operating stream and the applicationdata stream. This can cause problems within the application. Thereforepreferably any dual input data entering the application is only acceptedfrom the application data stream. The operating stream may be used forstandard operations within the application.

Preferably it will be possible for a user to configure each pen to be asystem pen; a private pen; both; or neither. A simple software utilitymay allow the end user to override the default pen configurations.

If both pointing devices are configured to be system pointing devices,each pointing device will behave like a standard pointing device on atypical whiteboard system (with the exception of the data rate). Theywill both be able to fully control the operating system and standardapplications running on it. They will interfere with each other if bothare placed on the surface at the same time. In this configuration, itwill not be possible to correctly control any dual applications.

One reason for a user wanting to set this generally undesirable mode ofoperation would be to allow them to increase the quantity of standard‘single use’ pens available within the classroom. However, it should benoted that the performance of each pen on the board will be half that ofa single pen in single pen mode, and this mode is therefore generallynot advantageous.

If a user configures both pointing devices to be multi-mode pointingdevices, they will only operate in a dual application. Neither pointingdevice will operate the operating system generally at the interactivesurface. In all configurations the physical mouse will continue tocontrol the operating system as normal. This configuration may beuseful, where the teacher wishes to lock two students into a dualapplication at the whiteboard, whilst maintaining system control usingthe physical mouse (e.g. whilst sat at the desk).

In another configuration, both are configured as system pointing devicesand multi-mode pointing devices, and can control the operating systemand a dual application. Again, the pens will interfere with each otherwithin standard operating system applications, and therefore this modewill likely have limited usefulness.

If both pointing devices are configured as neither system nor multi-modepointing devices then all pen input at the board is ignored. This may beuseful if the teacher requires to temporarily prevent students frominteracting with the board while the teacher performs some activity atthe desk using the mouse. If one of the two pointing devices isconfigured as neither, then all pointing devices of that nominalfrequency will be ignored. In this case the data rate of the remainingpointing device may be increased to the maximum value possible.

The optimum configuration is likely to be to configure one pointingdevice as a system and multi-mode pointing device; and one or moreothers as multi-mode pointing devices only.

Pointing devices are preferably provided with a default configuration ofone pointing device configured as a system and application pointingdevice, and one or more further pointing devices configured asmulti-mode pointing devices. An application package may be provided forthe interactive whiteboard system which enables a user to change theconfiguration of any pointing device. The system, under control of theprocessor, may dynamically configure the operation thereof in accordancewith the pointing devices configured by a user or in use. The user maybe able to define how many different pointing devices are to be used fora given application, and the processor can then dynamically adjust thefrequency generation for driving the drive grid in accordance with anappropriate number of frequencies. Where any pointing device is disabledat any time, the associated driving frequency can be disabled.

Support for more than one pointing device with the interactive systemrequires that:

-   -   a) the interactive whiteboard system is instructed to detect        more than one pen.    -   b) an interactive whiteboard system driver is instructed to        differentiate the data from each pen.    -   c) a dual pen aware application is configured to receive the pen        data from an interactive system device driver.

With reference to FIG. 6 the operation of a multi-mode configuredinteractive display system is further described. FIG. 6 illustrates themain software architecture for achieving multi-mode operation inaccordance with preferred embodiments of the invention.

Block 602 represents the interactive board. In a preferred embodimentthe interactive board incorporates all the system functionality of theinteractive operation as illustrated in FIGS. 2( a) and 4 as describedhereinabove. The interactive board 602 interfaces with other functionalelements of a computer system associated with the interactive system viaan interface 620. The interface 620 may be a USB, Serial, Bluetooth orother type of interface. The interface may thus be wired or wireless.

The interactive board 602 communicates with an interactive board devicedriver 604 over the interface 620. The interactive board device driver604 communications with a multi-mode API (Application Program Interface)606 over communications interface 622. The multi-mode API 606communicates with a multi-mode application 608 over communicationinterface 624. The interactive board device driver 604 communicates witha standard mouse driver 610 over communication interface 628. Thestandard mouse driver 610 communicates with a multi-mode application 608over communication interface 626. The communication interface 626 alsocommunicates with system applications. The interactive board devicedriver 604 is further connected to communicate with a multi-modeapplication 608 over communication interface 628.

The multi-mode application 608 represents a software applicationsupporting multi-mode use. In other words the multi-mode application 608is an application provided for a user, which has functionality whichallows two or more pointing devices to be used on the interactivesurface simultaneously. As will be understood by one skilled in the art,multiple such applications may be provided, and these applications maybe provided in combination with single mode applications. One multi-modeapplication 608 is shown in FIG. 6 for ease of explanation.

When installed, or activated, the multi-mode application 608 sends aninstruction, or series of instructions, to the interactive board devicedriver 604 via the multi-mode API 606 using communication interfaces 624and 622. The interactive board device driver 604 then sends any receivedinstructions to the interactive board 602. In embodiments where theinteractive board 602 may switch between functional modes, such receivedinstructions may cause the interactive board 602 to switch into‘multi-mode’. This may require, for example, the interactive board 602to switch into a mode where the drive grid is driven by multiplefrequencies. Depending on the multi-mode application 608 theinstructions sent to the interactive board 602 in this way may provideinformation as to the number of multiple pointing devices allowed to bein use at any one time. Where the user is configuring use of theapplication via the user interface, the user may indicate to theapplication the number of pointing devices to be used. The instructionsent by the multi-mode application may then indicate to the interactiveboard 602 the number of devices in use so that the appropriatefrequencies can be set.

Preferably the instructions sent to the interactive board 602additionally include an identification of the pointing devices that willbe in use. The identification of the pointing devices may then be usedby the interactive board 602 to identify the frequencies with which thedrive grid should be driven. Thus on the basis of instructions receivedfrom the multi-mode application 608 the interactive board 602 isconfigured to drive, and detect, a certain number of frequenciesassociated with a certain number of pointing devices.

In general once configured to operate in a particular mode of operation,and particularly to search for a particular number of frequencies, theinteractive board 602 will continue in this mode of operation untileither instructed otherwise or until there is a power-down operation.

The interactive board device driver 604 also responds to instructionsfrom the multi-mode application 608 as to the number of pointing devicesin use. The interactive board device driver 604 is also provided withinformation as to the type of such pointing devices, i.e. whether theyare system or multi-mode pointing devices (or both).

The interactive board device driver 604 is adapted in order to interpretand process data from each pointing device separately. The interactiveboard device driver 604 may interpret the data from one or more pointingdevice as:

(i) system data (from a system pointing device or mouse); (ii)application data (from a multi-mode pointing device); (iii) both; or(iv) neither.

Thus the interactive board device driver 604 receives data oncommunication interface 620 from the interactive board. In dependenceupon the pointing device from which such data is received, theinteractive board device driver will transmit that received data oneither communication interface 628 or communication interface 629. Datafrom application pointing devices is provided on communication interface628, and data from system pointing devices is provided on communicationinterface 629. As illustrated in FIG. 6 the multi-mode application 608receives the data associated with multi-mode pointing devices oncommunication interface 628 directly, and receives the data associatedwith system pointing devices on communication interface 626 from thestandard mouse driver 610. The standard mouse driver 610 may alsoprovide data from system pointing devices on communication interface 626to other system applications, such as associated with operating systeminteractions, rather than to the specific multi-mode application 608.

The multi-mode application 608 registers with the interactive boarddevice driver 604 in order to receive data from different pointingdevices in multi-mode operation. The interactive board device driver 604is adapted to store one or more registered application target addresses,such as a target address for multi-mode application 608, to whichapplication data received from application pointing devices is to besent to.

Preferably data received from the interactive board 602 on communicationinterface 620 and associated with application pointing devicescomprises:

i) a unique pen identification number (such as ‘pen 1’, ‘pen 2’);

ii) pen state (e.g. 0=no button pressed; 1=button 1 pressed); and

iii) the location position of the pointing device on the interactivesurface (i.e. x-y coordinates).

As mentioned hereinabove when a system pointing device is placed on theinteractive surface of the interactive board 602, the interactive boarddevice driver 604 will receive associated data and forward it to thestandard mouse driver 610 as occurs in normal interactive systemoperation (i.e. where only one pointing device is provided). Anyapplication, being a multi-mode application or otherwise, thereforereceives system pointing device data via the standard mouse driver as aresult of operating system events.

Data received by the interactive board device driver 604 from multi-modepointing devices is passed to all multi-mode applications which haveregistered with the interactive board device driver 604 for receivingthe application pointing device data. This data can be sent, forexample, via post messaging or by calling a call-back function withineach registered application.

In an alternative arrangement the interactive board device driver 604may store the data from any pointing device within a storage buffer, andprovide this data to any application upon request. A multi-modeapplication may, for example, poll for this buffered data.

Some applications utilising multiple pointing devices will now bedescribed. For the purposes of this discussion the pointing devices arereferred to as pens, although it will be understood that the principlesare not limited to pens.

All multiple pen applications are based upon the applicationprogrammatically making specific decisions about how to process theseparate pen event streams entering the application.

As in previous examples described above, in the following examples ofdual-pen applications are described for simplicity.

A first application is simultaneous input. A dual pen application can beprogrammed so that certain inputs can occur at the same time from two ormore pens. Simple examples are where both pens are annotating a documentdisplayed on the interactive surface; or one pen is annotating and onepen is moving (graphical user interface) GUI objects; or both pens aremoving GUI objects; etc.

A second application is assigning permanent functionality to pens. Anapplication can be programmed such that one or both pens are given apermanent function. A simple example is where one pen is permanentlyassociated with an eraser function and will therefore always erase anyannotation made on the interactive surface by the other pen. In thisinstance the ‘pen’ may be constructed such that it physically looks likea real board eraser.

A third application is assigning selectable functionality to pens. Anapplication can be programmed such that the user can specify aselectable function for each pen. A simple example is where a first penis set to annotate and a second pen is set to erase. At any time, theuser can choose other functionality for any pen by clicking on a GUItoolbox of available functions. For example, the second pen could bemade to behave as a highlighter by clicking the second pen on a‘highlighter’ GUI icon. The functionality of each pen can be stored andre-applied to the pens by the dual-mode application between sessions. Afurther example of this application is where each pen could be assigneda different language for the purpose of handwriting recognition. Forexample the annotations from the first pen could be always recognised asEnglish words, and those of the second pen as French words.

A fourth application is splitting the dual application input surfaceinto ‘pen specific’ active areas. A dual pen application can beprogrammed so that certain areas of the display associated with theapplication may be interacted with via either, both or neither pens.These areas may be any shape and quantity. In a simple form, a first pencould be configured to only work on the left half of the interactivesurface, and the second pen on the right half. This facility may promotethe development of dual applications of a competitive nature wherebyeach user is restricted to their own half of the surface in performingsimilar tasks.

This process involves the application holding a list of board areasagainst the pen identifiers. When a pen event enters the application,the application compares the pen's x-y position to decide if it lieswithin one of the areas associated with its pen identifier. If not, theevent is ignored otherwise the application can process the event.

This process could be generalised to areas of the operating system realestate by making the device driver process the areas, and restricting orallowing pen data accordingly. This particular function would onlyoperate with system pens and would only facilitate turn based clickactivities (not simultaneous pen movements).

There are potentially three ways of defining the active portion of theinteractive surface for a particular device. Firstly firmware can beused. The interactive display may have either fixed or downloadabledefined areas such that if the device is outside of its predefined areathe interactive system will not send any data to the host system. Asecond way uses the drivers. A driver could be intelligent in the sameway as the firmware solution by having predefined either fixed orloadable areas for each device, and only when a particular device iswithin its own operating area will the driver pass the information up tothe application. A third way is at the application level. The activeportions of the interactive surface may be defined via software. This isthe most advantageous and flexible way to provide the functionality. Theinteractive surface and the driver pass all the data for all of thedevices up to the application. The application then controls the area inwhich each pointer is active. The advantage of this approach is that itoffers much greater flexibility in defining areas, and these areas canbe much more easily changed ‘on the fly’ dependent upon changes withinthe application or application switching.

A fifth application is pen specific GUI elements. A dual pen applicationcan be programmed so that certain GUI interface elements are associatedwith either, both or neither pens. A simple example of this is aninterface button being associated with the second pen. In this exampleonly the second pen is able to click on the button, whilst use of thefirst pen on the button is ignored by the application.

Further examples include creating pen specific GUI toolboxes andactions.

A sixth application is one pen controlling the operation of the other. Adual pen application can be programmed so that the operation of thesecond pen is dependent upon some action performed by the first pen (orvice versa). A simple example is where the second pen does not functionuntil the first pen is used to click a GUI element (e.g. a buttonentitled ‘enable 2^(nd) Pen’). This promotes the idea of theteacher-pupil scenario where the teacher can decide when to allow thepupil (with the second pen) to interact with the dual application bytoggling the ‘enabled state’ of the second pen via a GUI element withthe first pen.

A seventh application is GUI tasks performed by combined pen actions. Adual pen application can be programmed so that certain operations can beperformed on GUI objects contained within the dual pen application whenusing both pens together.

Examples of these are:

-   -   Rotating an object. Place both pens on the same GUI object (e.g.        an image) and rotate the one or both pens physically about each        other to alter the angle of the underlying GUI object by the        same physical increment angle traced by the pens.    -   Steering an object. Place both pens anywhere on the same GUI        object (e.g. an image) and move the pens in the same general        direction. The object will follow the movements of the pen and        will be rotated according to the relative angle between the two        pens in order to ‘steer’ the object into a specific location and        orientation.    -   Aspect sizing an object. Place both pens anywhere on the same        GUI object (e.g. an image) and move one or both pens closer        together or further apart. The underlying GUI object will alter        in size by an amount which is directly proportional to the        physical change in distance between the two pens.    -   Stretching an object. Place both pens approximately on opposite        edges (or corners) of the same GUI object (e.g. an image) and        move one or both pens closer together or further apart. The        underlying GUI object will alter in size and aspect as dictated        by the change in pen positions.    -   Splitting an object in two. Place both pens near each other and        near the same edge of the same GUI object (e.g. an image) and        move one or both pens further apart. The GUI object will be        split into two GUI objects, the split point being defined as the        point on the edge of the original object that is midway between        the starting positions of the two pens.    -   Joining two objects. Place each pen on a different GUI object        (e.g. two images). Use the pens to bring two edges of the two        GUI objects within close proximity of each other. Release the        pens. The two GUI objects are joined together as one GUI object.    -   Connecting two objects. Place each pen on a different GUI        object. Without removing the pens from the board, drag the two        objects together so that they partially overlap, then drag them        apart again. This action can be detected by the application and        the two objects can then be shown connected (e.g. via a line        being drawn between them). Similarly, two connected objects can        be disconnected by repeating this action.    -   Duplicating an Object. Place the first pen on one GUI object and        do not move it. This object is now classed as anchored (or        pinned) in place. Place the second pen on the same GUI object        and drag the second pen away from the first pen in order to        introduce a duplicate copy of the pinned object which is then        dragged by the second pen.    -   Cropping an object. Place both pens on the same GUI object (e.g.        an image). The two pens form the opposite corners of a cropping        rectangle which can be indicated on screen. The pens can then be        moved in, order to adjust the cropping rectangle. The cropping        function can then be performed by clicking either pen on another        GUI element or simply by removing the pens from the board        surface.    -   Measuring Distance. Place both pens on different points in an        application. The distance between the two pens can be calculated        based upon the current scale factor of the application.

An eighth application is application pen interaction over the operatingsystem. A dual pen application can be programmed such that wheneverapplication pen data is received, this can be displayed in a transparentwindow over the operating system. This allows for one person to use asystem pen in order to interact with any standard application (e.g.Word), whilst the user of the second pen uses application pen data tosimultaneously annotate (or manipulate other GUI elements) over the topof the standard application without interfering with the activity of thesystem pen.

A ninth application is ‘wrap around’ desktop. Users of the dual modeinteractive system may stand at either edge of the board allowing thecentral projected portion of the board to be clearly visible to theaudience. In order to facilitate sharing GUI elements (e.g. toolboxes),a first person (with a first pen) at the left hand edge of the desktopis allowed to drag a GUI object off the left hand side of the desktopand for it to appear at the right hand side of the desktop (or viceversa) for use by the second person (with a second pen).

A tenth application is throwing objects. Additional to ‘wrap around’desktop is the idea for a first person (with a first pen) at either sideof the interactive surface to click and drag on a GUI object (e.g. atoolbox or colour palette) and to describe a ‘quick flick’ motion in thedirection of the opposite side of the board in order to transfer theselected GUI object across the screen surface to the other edge of theboard, in order that it can be used by a second person.

An eleventh application is automatic left/right docking. If a GUIelement is associated with a specific pen, the application can track theposition of the pen and can alter the docking of the GUI elementaccordingly. One example is where both pens each have their own toolbox,docked at the left and right hand edges of the interactive surfaces. Ifthe users switch sides (i.e. they both physically walk to the oppositesides of the board), the application can detect that the pens are nowgenerally being used on the opposite halves of the board and cantherefore automatically switch the docked toolbars to the opposite sidesof the board.

A twelfth application is interacting with general standard applications.When configuring both pens to be system pens, the device driver can thenbe instructed to operate in a mode where it remembers each pen positionand button state, such that it can generate, interleave and sendstandard mouse events to the mouse driver for both pens. This involvesinserting artificial mouse-up-move-down events between data packetsgenerated by sequential multiple pen data arriving at the device driver.This approach will allow both pens to simultaneously operate differentparts of the same standard application, or for each pen to operate adifferent standard application.

This allows the majority of drawing and standard interface GUI objectsto be interacted with simultaneously with both pens.

One example is that a first pen may be used to scroll the verticalscrollbar and the second pen may be used to scroll the horizontalscrollbar of any application, simultaneously. A second example is thatboth pens may be used to draw and move objects around any standardgraphics application simultaneously.

In accordance with further embodiments of the present invention theabove-described interactive display system is adapted to allow foradditional functionality.

Referring to FIG. 7, there is shown a portion of the interactive surface704 having an upper surface 710 which in use may comprise the worksurface and a lower surface 712 which in use may be mounted against awall. In use a pointing device such as pen 708 is moved around the worksurface to provide operation similar to a mouse device as discussedhereinabove. The pen contacts the surface 712 in order to interact withthe interactive surface and provide inputs.

Above the work surface there is provided a region 702 which extends fromthe surface 712 to a distance as denoted by dashed line 702. This is aregion in which an interaction still occurs between the pen 708 and theinteractive surface 708, such that the presence and movement of the pen708 may be traced. This region 706 may be referred to as a proximitylayer or a layer of interaction above or along the interactive surface.This layer of interaction is just above the surface of the board, to aheight of approximately 8 mm.

The pen 708 may still move and be detected in this layer of interaction,to move a displayed cursor, without the pointing device touching thesurface. The principle of position sensing of the pointer device withinthe proximity layer is known in the art.

The pointing device may be used to describe one or more ‘gestures’within the interactive surface proximity layer. These described gesturesmay be then recognised, in accordance with predefined principles, andtheir detection used to invoke predefined actions within an applicationrunning on the interactive system. This operation may be performedwithout interfering with the underlying displayed image material becausethe pointing device does not physically touch the interactive surfacewhilst making the gesture. Thus a distinction is made between movementof the pointing device to control an interaction with an application andcause the displayed image to change, where the pointing device mustphysically touch the interactive surface, and movement of the pointingdevice proximate but not in physical contact with the interactivesurface, which preferably does not cause the displayed image to change.

This functionality is described in the context of an exemplaryinteractive system which utilises electromagnetic interactive surfaces.It cannot be achieved with touch sensitive boards. The functionality canalso be achieved, however, with optical technology. The use of gestureactions provide a user with a significant productivity benefit.

By way of example, a gesture might be used to switch tools from a pentool to a marquee select tool. A further distinct gesture might call upa colour selection panel. Using just these two gestures anywhere overthe interactive surface allows the user to annotate in any colour andmove the annotations around without ever requiring the user to accessthe main toolbox.

Gesture actions may be disabled by default, and be enabled andconfigured within settings as discussed further hereinbelow. A number ofgestures are preferably predefined.

In use, each gesture is made by moving the pointing device from anotional central starting point, out to one or more notional secondarypoints (which may be referred to as gesture points), and finishing backat the centre point. The gesture thus essentially traces the pointingdevice over a number of dots within close proximity.

The gestures can be understood by way of examples. The gestures can beunderstood if it is imagined they are made from the centre of animaginary compass. In an example eight gesture points are providedplaced equally spaced around a start point. The gesture points may thusbe considered to be at positions North, North East, East, South East,South, South West, West and North West of the start point, as shown inFIG. 8( a).

Further by way of example, the gesture points may be labelled withreference numerals such that the ventral point is 0, North is 1, NorthEast is 2, East is 3, South East is 4, South is 5, South West is 6, Westis 7 and North West is 8. This is shown in FIG. 8( b).

In the Figures, each gesture point is shown as surrounded by a circle.This circle is the preferred gesture tolerance area into which a gesturetrace must pass in order for the associated gesture point to beacknowledged as being reached.

This ‘point tracing’ technique offers many combinations of gesture.

There are eight possible short line gestures (or so-called flickgestures) where the user moves the pen quickly out to one of the eightgesture points and back to the centre point. For example the user startsfrom the centre, flicks the pen up to the North gesture point, andflicks the pen immediately back to the centre, providing an up-downgesture. This is illustrated in FIG. 9( a). FIG. 9( b) shows a shortline right-left gesture.

Each gesture can be defined by a sequence of numbers, using theannotations of FIG. 8( b), describing the order of the points that makeup the gesture. In the above examples the number sequences are ‘10’ forFIG. 9( a) and ‘30’ for FIG. 9( b). The initial ‘0’ need not be includedin the number sequence as all gestures preferably include this startingpoint by default.

There are eight possible long line gestures. This is where the pen movesout to any gesture point, then back through the centre point to theopposite gesture point, finishing back in the centre. Examples are shownin FIGS. 10( a) and 10(b).

There are eight possible double short line gestures. This is where thesame short line gesture is made twice in succession (the ‘doubleflick’). Examples are ‘8080’ and ‘5050’.

There are many line combination gestures which may be defined. Oneexample is to form a right angle between the centre point and two othergesture points. In the example of FIG. 11, the pen is moved North, thenback to the centre, then West, then back to the centre.

There are many possible shape gestures. This is where the pen is movedout to one of the eight gesture points, and then moved to one or moreother gesture points before returning to the centre point.

One example of a shape gesture is a quadrant gesture. There are manyquadrant gestures. FIGS. 12( a) and 12(b) show two examples.

Another example of a shape gesture is a pie slice gesture. There aremany possible slice gestures. Examples are shown in FIGS. 13( a) and13(b).

Another example of a shape gesture is a semi-circular gesture. Againthere are many semi-circular gestures. Examples are shown in FIGS. 14(a), 14(b), 14(c), and 14(d).

Another example of a shape gesture is a circular gesture. Circulargestures may be provided which require the user to trace through all thegesture points and back to the centre as illustrated in FIG. 15.

In practice gestures of this length and accuracy are difficult to traceand hence may not be provided for.

For all shape gestures it is possible to draw them in one of two ways.For example the first ‘quadrant gesture’ shown in FIG. 12( a) may bedescribed by tracing through the points ‘1230’ or ‘3210’. In allinstances the program will preferably treat these as unique gestures.

It will be understood from the preceding discussion that there is analmost limitless number of gestures that can be defined. However, therange of gestures preferably needs to be kept fairly small so that theuser can learn and remember them all.

As discussed previously, any gesture is made within the proximity layerand may be best performed whilst the users hand is resting on thesurface of the board.

A fundamental principle that all gestures have in common is that theystart at the centre point and end at the centre point. This rule initself helps the program to avoid detecting random gestures out of thenormal movements of the pen over the surface of the board.

A new gesture actions settings panel will preferably be provided asshown in FIG. 16. A test area allows the user to practice gestures. Itdisplays the eight possible gesture points surrounding a notionalstarting centre point. Each gesture point is shown with a tolerancecircle whose radius is defined by a ‘tolerance’ slider. The bigger thetolerance circles, the less accurate the gesture can become.

The distance between the centre point and the gesture points is definedby a ‘size’ slider.

The ‘gesture list box’ contains a list of sixty four possible gesturesthat the gesture engine can recognise. The first gesture in the list is‘10’. This is the ‘up-down’ gesture. To perform this gesture, the usermoves the pen to the central cross, then quickly flicks the pen up togesture point 1 and back to the starting point (point 0) withouttouching the board, ensuring that the target point is reached during themaking of the gesture.

The ‘action list’ shows the currently selected action for the currentlyselected gesture. One of nearly 200 actions can be associated to each ofthe 64 gestures. The list of actions may be taken selectively from thestandard functions and page and object actions of any application beingused.

The starting point for a gesture can actually be anywhere on theinteractive surface. The settings test centre point is only provided asa guiding start point.

The speed at which the gesture must be made can be altered with the‘speed’ slider, together with the size and tolerance of the gesture.

In the example of FIG. 17 it can be seen that the user has increased thetolerance circles and made the required gesture size much larger. It canalso be seen that the double short line gesture ‘down-up-down-up’ (5050)has been set to show the Colour Selector panel as illustrated further inFIG. 17.

When gesture actions are enabled they will preferably only operate whenperformed over the surface of the interactive surface, rather than onthe surface. Preferably the gesture actions are enabled only whenperformed over an active portion of the interactive surface associatedwith an application.

For the example of a flipchart application, the active portion of theinteractive surface may be the portion on which the work surface of theflip chart is displayed. Any one of the gestures can be made anywhereover the surface of the flipchart at any time in order to invoke theassociated action. Alternatively the gesture actions may be enabled whenperformed anywhere over the entire surface of the interactive surface,irrespective of any running application. In such circumstances thegesture may enable an action associated with the operating system.

If a ‘Show Gesture Points’ setting is enabled, in use in an applicationthe user may see the gesture target points drawn as light grey pixelsaround the pen position on the displayed image on the interactivesurface.

Whilst the invention and the various embodiments thereof is ideallysuited to passive electromagnetic positioning technology as describedherein, it may also be used with active pointer devices. The inventionand its embodiments are also not limited to electromagnetic interactivetechnology. Optical technology may also be used.

It should be noted that although the description herein is presented inthe context of an interactive display system incorporating a whiteboardassembly arrangement, the invention is not limited to such. Theinvention generally applies to interactive input/output devices, whichmay include, for example, graphics tablets such as may be used ininteractive display systems. The invention may generally be used withany type of interactive display adapted to interact with a pointingdevice.

The invention claimed is:
 1. An interactive display system comprising aninteractive surface for displaying an image, the interactive surfacebeing adapted for detecting the positions of a plurality of pointingdevices relative to the interactive surface, wherein one pointing deviceis configured to control the operating system and at least one furtherpointing device is configured to interact with an application.
 2. Theinteractive display system according to claim 1 in which data receivedfrom at least one pointing device configured to interact with anapplication displayed in a transparent window.
 3. The interactivedisplay system of claim 1 wherein the at least one further pointingdevice is for interacting with an application which is configured formulti-pointing device use.
 4. The interactive display system of claim 3wherein the one pointing device is further adapted to interact with oneor more applications.
 5. The interactive display system of claim 3wherein the one pointing device can be used in any standard applicationand applications supporting multiple pointing devices.
 6. Theinteractive display system of claim 3 wherein the application pointingdevice can be used to interact only with applications supportingmultiple pointing devices.
 7. The interactive display system of claim 3wherein a pointing device is configurable as a system pointing device oran application pointing device by a user.
 8. The interactive displaysystem of claim 3 wherein the interactive display system includes aninteractive board device driver adapted to interpret and process datafrom each pointing device separately.
 9. The interactive display systemof claim 8 wherein data is interpreted as system data or applicationdata.
 10. An interactive display system according to claim 8 wherein anapplication adapted to detect multiple pointing devices receives thedata associated with the application pointing devices on a communicationinterface directly, and receives data associated with a system pointingdevice on a communication interface from a mouse driver.
 11. Theinteractive display system according to claim 8 wherein a mouse driverprovides data from a system pointing device to a system application. 12.The interactive display system of claim 8 wherein one user can use asystem pen to interact with any standard application whilst another usercan use an application pen in order to simultaneously annotate ormanipulate over the top of the standard application.
 13. The interactivedisplay system of claim 1 comprising a drive grid signal generator fordriving a drive grid for a period of time at each of a plurality offrequencies associated with each of the plurality of pointing devices,wherein the system adapts the period of time for a given frequency tooptimize said period of time for particular devices.
 14. A method in aninteractive display system comprising an interactive surface fordisplaying an image, the interactive surface being adapted for detectingthe positions of a plurality of pointing devices relative to theinteractive surface, the method comprising configuring one pointingdevice to control the operating system and at least one further pointingdevice to interact with an application.
 15. The method of claim 14wherein data from an application pointing device is transmitted on onecommunication interface.
 16. The method of claim 15 wherein data from asystem pointing device is provided on another communication interface.17. The method according to claim 14 wherein data received from thesystem pointing device is forwarded to a mouse driver.
 18. The methodaccording to claim 14 wherein data received from an application pointingdevice is passed to all applications adapted to detect multipleapplication pointing devices.
 19. The method of claim 14 wherein datafrom each pointing device is processed separately.
 20. The methodaccording to claim 14 further comprising driving a drive grid for aperiod of time at each of a plurality of frequencies associated witheach of the plurality of pointing devices, and adapting the period oftime for a given frequency to optimize said period of time forparticular devices.